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K f % . eVeiluation consultation services to all local educational agencies 
((grades K- 14) in 'Los Angeles County. The three prpgrai evaluatiofi 
: services that are described are: (1) Educationai Prograi JEvaluatibn 
Planning Assistancer (2) Educational Progra* Audit Assistance, and 
^ v (3) Direct Evaluator Assistance. (One section of the pape^^ 

an; introduction to the audit process through the use of a , 
c ; questipn-and-ansiier foriat) . The program evaluation services have 
recently provided to lore than 40 eleientary, secondary, and 

: cbBlluni,ty college agencies. In addition, the progress or results of , 

evaluations of: Early Childhood, Coipensatory Education, Hentally , 
- Gifted Hinors, Educationally Handicapped, Eansoi Beading, Career 

AVaren®^s, CoMunity Colleg® Veterans* Becruit«ent, an^ CoMunity 
College Children* s Centers will be described. The asiserti on will be 
■adc that behavioral objectives as used in prograi evaluations are 
bssentiariy research hypotheses and that data analysis and 
, Interpretations should recognize this through the eiployient of 

statistically derived confilence intervals. Another assertion that 
will be made is that three criteria should be used in judging progra* 
evaluation: (1) standards of bebavioral science research; (2) 
utility; and (3) cost-benefit considerations. Finally, two progra« 
evaluation sodels will be coipared.to the one that has evolved in the 
^^^^^^ : C^ 



II) 104 919 

TITLE 



POB iJATE 
HOTE 



EDBS PBICE 
DESCBXPTOBS 



IpEHTIPIEfiS 
ABSTBACT 




CJN 



o 



THE STATE- OF THE ART & SCIENCE OF EDUCATIONAL PROGRAM- 
EVALUATION SERVICES (K^14) IN LOS ANGELES COUNTY AS 
EXPERIENCED BY ONE LOS ANGELES COUNTY CONSULTANT 



Q 



O S.OePARTMENTOFHEAUTM. 
EDUCATION A WFLFAHE 
NATIONAL INSTITUTE OF 
EDUCATION 
DOCUMENT XA3 eGGN RgPRO 
0\fCeO EXACTLY AS RGCEWGO f=^ROM 
Iht PERS0NrORO«t?ANl2AT!ON ORIGIN 
AUNG IT P08NT5 O*^ VitlWOR OPINIONS 
SIATEO 00 N0\ NLCLSiARlLY REPRE 
SCNTO^^flCSAVNATlOSAL mSUTUTE OF 
XO\}CATi(jfi Post T. ON OR POL*CV 



BY: 



DONALD.L. KESTER, PH.D.. - • ^ 
CONSULTANT 

EDUCATIONAL PROGRAM AUDIT AND EVALUATION 
LOS ANGELES COUNTY SUPERINTENDENT OF SCHOOLS OFFICE 
9300 EAST IMPERIAL HIGHWAY 
DOWNEY, CALIFORNIA 




1^5 



ii 1 tt HitJimfi 



PRESENTED AT THE 
197A ANNUAL CALIFORNIA 
EDUCATIONAL RESEARCH ASSOCIATION 
CONFERENCE IN SAN mNCISCO 




NOVEMBER, 1974 



ERIC 



2 



TABLE OF COimMTS 



Abstract. 

Section 1 : Introduction and Acknov lodgoMntn « 



Section 2: Definition and Itolo of the Threr Prograa Evaluation 

Services nw Offered by the Los Angeles County 



Section 3: A comparison of the UCLA Evaluation Model » the CIPP 

Evaluation Model, and the Evolving Los Angeles County 
Evaluation Services Model. • . . . « 



Section 4: Progress or Results of Los Angeles County Evaluation 
Services for Selected LEA Programs ..«......•«....•..« 



Section 5: Program Evaluation Performance Objectives are Essentially 
Kesearch Hypotheses. . ....... . . . . . . . «... • . . • • . . • . . . . . 



Section 6: Sowe (^lestions and Im^ the Educational 

Program Audit Process. ....... .......... ••••••••••••••••• 

Section 7: Conclusion. • . • • • • • • 

Bibliography. . . ....... • - • • • • • • • . . • * • • • • • • • • • • • • • • • • • • 

Appendix A: UCIA Center for the Study of Evaluation, Evaluation Model... 
Appendix B: Local Educational Agencies in Los ^geles County , 



'...........4 



Appendix C: Administrative Off leers of the Los Angeles County 
Superintendent of Schools Office. 



Appendix D: Programs for which Services have been Provided by ^ 
Consultants in Educational Program Evaluation and 
Educational Program Audit . . . . . • . . . ... ......... . . . . . ...... • • * 

Appendix E: Alphabetical Listing of Los Angeles County Evaluation 
ConsultiEints 



s 



1 



The Division of Program Evaluation and Research of the Los Angeles County * 
Superintendent 's Office provides program evaluation consultation services to all . 
local educational agencies, (grades K-IA) in Lbs Angeles County. Since this county 
is the most populous one in the State (it has more than A 5% of the State's school 
age young people) » program evaluation services have been provided for a great many 
different kinds of educational progrsms with a variety of outcomes. The three . 
program evaluation services that are described are: (1) Educational Program Eval- 
uation Planning Assistance, (2) Educational Program Audit Assistance, and (3) Direct 
Evaluator Assistance. (One section of the paper presents an introduction to the 
audit process through the use of a question-^and-answer format.) The program eyal- 
uatioh services have _ recently been provided to more than AO elementary, secondary, 
and community college agencies. In addition, the progress or results of evaluations 
of: Early Childhood, Compensatory Education, Mentally Gifted Minors, Educationally 
Handicapped. Ransom Reading, Career Awareness, Community College Veterans' Recruitment, 
and Community College Children's Centers will be described. The assertion will b^ 
made that behavioral objectives as used in program evaluations are essentially 
research hypotheses and that data analysis and interpretations should recognize 
this through the employment of statistically derived confidence intervals . Another 
assertion that will be made is that three criteria should be used in judging program 
evaluation: (1) standards of behavioral science research;^ (2) utilityj and 
(3) cost-benefit considerations. Finally, two program evaluation models will be 
compared to the one that has evolved. in the County Off Ice. 
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"THE STATE OF THE ART & SCIENCE OF. EDUCATIONAL PROGRAM 
EVALUATION SERVICES (K-14) IN LOS ANGELES COUNTY AS 
EXPERIENCED BY ONE LOS ANGELES COUNTY CONSULTANT 



Se ction 1; Introduction and Acknowledgem ants 

The purpose of this presentation Is to give you one person's view of the State 
of the Art and Science of Educational Program Evaluation Services provided by the^ 
Los Angeles County Superintendent of Schools Office to the 82 school districts and 
20 cciraminlty colleges In Los Angeles County. You may be Interested to hear that 
these Educational Program Evaluation Services are provided upon their request and 
are free of charge to these 102 local educational agencies. 

There are three different kinds of Program Evaluation Services that are pro-^ 
vlded. These three are called: (1) Educational Program Evaluation Planning Assistance 
(2) Educational Program Audit Assistance,' and (3) Direct Evaluator Assistance. At 
present the Los Angeles County Office has twelve full-time consultants who provide 
these three kinds of Program Evaluation Assistance. 

The County Off Ice has not always provided this kind of evaluation assistance. 
In fact, this 3-way delivery of evaluation assistance is quite recent. After the 
idea of offering Educational Program Audits to requesting LEA' s evolved, there was 
initially a period of pilot testing that took place. VTlthin Dr. Maylon Drake's 
Administrative Service Area, Dr. Jean Wiener. Dr. Ruth Cohen, and Dr. BlU Turner 
undertook an J'ln-house audit^' of an educational program that was being directed by 
other consultants who worked at the Los Angeles County Office.' T^ile '*in-house 
audits" are not strongly recommended, the experience gained was valuable. After 
the "In-house audit'* the Educational Program Audit Service was offered to six 
selected districts. Since that time requests for audits increased to 18 for last 
vear a973-74) and 30 for this year (1974-75) . 



-Those who helped Hrinjx about the evolution of these services should be 
achnowledr,ed. These "change agents*', or "prime movers", include Dr. Richard 
Clowes, Superintendent; Dr. Maylon Dralte, Assistant Superintendent for Educational 
Proj^rams and Services: Dr. Gordon Footman, Director, and Dr. Warren Newman, 
Assistant Director, of the Division of Program Evaluation, Research, and Pupil 
Services; Drs. Jean Wiener: Ruth Cohen; and Bill Turner. The members of the 
Los Angeles County Board were also supportive of this effort; they are: 

^Dr. Daniel L. Towler, President: Vincent H. Simpson, Vice President; Robert M. 
Bock, Member; Mrs. Pauline Chace, Member; Mrs. Lucille M. Fields, Member; 
Dr. R. V. Pullias, Member; and Mrs. Kathryn Vanderhook, Member. 

Needless to say, without the efforts of all these people the three-way 
Evaluation Services Delivery System would not have come Into being i 

* ; W^^^^ detailed description of the evolution of the Three Evaluation 

Services might be interesting to some, it suffices to say here that the three 
distinct services previously mentioned: (1) Evaluation Planning, (2) Prpgtam 
Audit, Jind (3) Direct Evaluator Assistance are the three services that are now 
being provided. Of course, like all organizations, the Los Angeles County Office 
is in a period of transition, but for the time being and for the foreseeable future 
these three services seem to have achieved a measure of stability. 

Section 2; Definition and Role of the Three Program Evaluation Services 

The Three Evaluation Services are: (1) Educational Program Evaluation 
Planning Assistance, (2) Educational Program Audit Assistance, and (3) Direct 
Evaluator Assistance. These three services are defined as follows: 
T. Educational Program Evaluation Plannin g Assistance 

Local educational agencies may elect to receive Educational 
: Program Evaluation Assistance prior to having an audit of a prx)gram. : 
Educational Program, Evaluation Assistance is a service which is 
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Intended to strengthen the internal process of evaluation in a 
local educational agency* In collaboration with a county consultant, 
the local evaluator designs an effective evaluation system with 
appropriate statistical procedures which includes development of 
performance objectives, process objectives, evaluation specifications, 
the evaluation design^ and. rd^ated documents. 

II. Educational Program Audit Ass istance 

Local educational agencies may elect to have an educational 
program audit. The educational program audit Is a performance 
control process based upon external reviews conducted by qualified 

y~^-o\it3ide consultants. It is designed to verify^ the results of the 

evaluation of an educational program and to assess the appropriateness 
of evaluation procedures used for determining the effectiveness of the 
operation and management of the program. 
III. Direct Evaluator Assistance 

An evaluator at a local educational agency may elect to receive 
direct evaluator assistance in the development of an evaluation 
management plan, statistical data analysis, data synthesis, and 
evaluation report writing and. interpretation. 



. In chemistry it is known that a set of reactants can be made to produce different 
products. The products that are produced depend on the catalyst used and the experi- 
mental condition chosen. For example, carbon monoxide and hydrogen can produce 
either methane (CH^) or methanol, depending on the catalyst used. 

An analogy exists between a chemical catalyst and the role of County Office 
Evaluation Consultants. It is intended and expected that these three services act 
somewhat like a catalyst to help. Improve evaluation skills in Los Angeles County. 
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Sec tio n -3; A Ccn.p.r<«nn nf the U.C.L.A/ Evaluat ion Model^tJieAailtP^ 
Mftdel , and the Evolving Los Angeles Connty Eva luation Services Model 

The Program Evaluation Services Model that has evolved at the Los Angeles County 
was developed out of a face-to-face relationship with the LEA personnel and does not 
exactly and completely conform to either the U.C.L.A. Evaluation Model or the C.I .P. P. 
Evaluation Model. However, there are strong similarities between them. 

The U.C.L.A. Center for the Study of Evaluation has definied evaluation and 
listed five evaluation phases. The Center defines evaluation as "....The process of 
determining the kinds of decisions that have to, be made and selecting, collecting, 
and interpreting the information needed in making these decisions." The five 
evaluation phases are: Needs Assessment, Program Planning, Implementation Evaluatipn, 
Progress Evaluation, and Outcome Evaluation. The definition of each of these five 
phases Is shown in Table 1. (Center for the Study of Evaluation, Evaluation 
Workshop I, 1971.) • ; 
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Phase Ir Needs Assessment Involves stating the objectives to be met 
and determining hot7 well an existing program is meeting 
these objectives, this information is used to identify 
school or program needs. ' - 

Phase 2: In Program Planning, the evaluatcr provides the project 
: ^ director with ^.tools to help make planning decisions. - He 

also builds into the program the procedures that will be 
needed for assessing whether or not it is operating as 
planned and how well it is achieving its objectives. : 

Phase 3: Implementation Evaluation is a monitoring process to d<eter- 
mine the extent to which the specified elements of the 
program have been implemented as planned. 

Phaise 4: Progress Evaluation provides information about the progress 
of the program's objectives. This information i> used to 
tnodify the program where necessary. - - 

Phase 5: Outcome Evaluation provides information about the success 
of the entire program. This information can support a 
decision to maintain, modify, expand, or discontinue the 
program. 




stuff lebeam, the Director of the Western Michigan University Evaluation 
Center, has defined evaluation as ". . . the process of ascertaining the relative 
values of competing alternatives ... (it is the process) . . . of supplying 
information for decision makers." According to Stufflebeam's C.I.P.P. Model of 
Evaluation there are four kinds of evaluation that are concerned with four classes 
of evaluation decisions and they focus concern on four different areas. These are 
shown in Table 2 (Stuff lebeam, Daniel.. "Use of Experimental Designs in Educa- 
tional Evaluation," 1971). 

TABLE 2 - Kinds of Evaluation, Classes of Evaluation Decisions, and Focus 
of Concern of the C.I. P. P. Evaluation Mbdel 



1 _ . . 

The 4 kinds 
of 

Evaluation 


The 4 classes j 
of Evaltiation 
Decisions 


Focus of Concern 


Context 


Planning 


Identifying unmet needs » selecting program 
objectives - - 


Input 


Structuring 


Projecting and analyzing alternative pro- 
cedural designs ^^to achieve the* stated 
objectives 


Process 


Implementing 


Operationalizing and executing the program 
design, monitoring program operations 


Product 


Recycling 


Judging and reacting to program results. 

Ascertaining the degree to which objectives j 

. _ . . . - . . t. 

have been attained 



A tentative program Evaluation Services Model ha6 been developed by 
Los Angeles County staff* This Flowchart Model is shown below in Display 1* 



DISPLAY.^1 



Page, 7 



EDUCATIONAL PROGRAM EVALUATION 
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(f) Develop the Budget 
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From the descriptions of both the U.C.L.A. and C.I. P. P. Models and the 
descriptions of the three Los Angeles County Services, It seems clear that the 
six Los Angeles County Educational Program Evaluation Planning Consultants work 
with LEA personnel In Phase 1 (Needs Assessment) and Phase 2 (Program Planning) . 
of the U.C.L.A. Evaluation Model as well as with the Context and Input areas 
of the C.I. P. P. Evaluation Model. It further seems clear that the four hoa 
Angeles County Educational Program Audit Consultants work with LEA personnel In 
Phase 3 (Implementation Evaluation) , Phase A (Progress Evaluation) and Phase 5 
(Outcome Evaluation) of the U.C.L.A. Evaluation Model, along with Procpss and 
Product areas of the C.l.P.P. Evaluation Model. In the past, the two Los Angeles 
(iounty Direct Evaluator Consultants have probably worked more In Phase 5 (dutconie 
Evaluation) of the U.C.L.A. Evaluation Model and area 4 (Product Evaluation) 
of the C.l.P.P. Model. But, with the recent development of the Planning and 
Audit Services, a real effort has been made to Involve the Direct Evaluator 
Consultants earlier; that Is, In Phase 3 (Implementation Evaluation), Phase A 
(Progress Evaluation), and Phase 5 (Outcome Valuation) of the U.C.L.A. Evalu- 
ation Model and the areas of Process and Product of the C.l.P.P. Evaluation Model 

Section 4 - Progress or Results of Los A ng eles County Evaluation Serv ices for 
Selected LEA Programs * 

During the 1973-74 school year, audits were conducted for eighteen LEA 
programs. Among those were audits of: Early Childhood Educational Prop ^am.s. 
Title 1 Programs, A Mentally Gifted Minor Program, a Career Awareness Program, 
and a Ransom Reading Program. Some of the objectives of these programs are shown 
In Displays 2 through 6*. (The credit for the format of the audit plans goes to 
Dr. Jean Wiener, Dr. Ruth Cohen, and Dr. Bill Turner.) 



*Tb preserve, confidentiality, the local educational agencies are not named. 
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During the 1974-75 school year, an Educationally Handicapped Program and a 
Community College Veterans' Recruitment Program are being audited. Also during 
this year a Cptnnunlty College Children's Center Ptogram Is receiving Program 
Evaluation Planning Assistance. Most oj the objectives of the Community College 
Children's Center Program have been developed. Some of the objectives of these 
programs are shown In Displays 7 through 10. 
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- OUTCOME OBJECTIVES FOR THE VETERANS* OUTREACH PROGPAM 




Total Program Product or Outcome Objectives ; 



KG 



Full-tlnrc benefit receiving 
veterans' enrol lment (Ful l- 
thne -means taking 12 units 
each semester) 



—will Increase S% per year. 



2.0 

At least 3 veterans 



-will attend at least one 
Advisory Committee meeting 
by December t I97^« 



3.0 

At least 15 veterans with- 
out high school diplomas '-«wlll attend remedial sessions 

or classes at Harbor. 



k.O 



Box of veterans receiving 
benefits 



—will see veterans* office 
personnel at least twice 
per semester. 



5-0 

At least 500 veterans 

receiving benefits —will complete a minimum of 

> * 6 units In a given semester. 
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DISPLAY 10 - SELECTED OBJECTIVES FOR THE COMMUNITY COLLEGE CHILDREN'S CENTER PROGRAM 



GOAL 1 To provide exe«pUry child development prorrasi and lervicti that foster the 
cognitivep iocialt enotional/ and physical development for each child 
participant. 



TARGET 



TASK 



MEASURE 



DATE 



1.0 The Director 
and staff 

2.0 The Director 



S«0 The Eva^luator 



A.O The Director 
and staff 

S.O The Coordinator 



6«0 The Evaluator 



11.9 The Coordinator 



12*0 The Evaluator 



Will hold curriculum 
planning meetings 

Will keep at the center 
minutes of all staff 
meetings 

Will examine meeting 
minutes 

Will provide interest 
centers 

Will visit the center 



Will review Coordinmt 
tor*s checklist 

Will inform the 
Director and Evaluator 
in writing of all 
regulations and 
ordinances pertaining 
to the center 

Will ensure that the 
center is in compliance 
with all ordinances and 
regulations 



Minutes in Director U 
office. 

Minutes in Director's 
office. 



Evaluator* s report in 
evaluator* s office. 

Coordinator's checklist 
in Coordinator's office. 

Coordinator's checklist 
in Coordinator's office. 

^valuator's report in 
nvaluator's office. 

Coordinator's copy of 
Tiemo, letter, and* manual 
In Coordinator's office. 



:valuator's report and 
:heckliat in ^valuator's 

>ffice. 



Each week 



During semester 



Within two weeks 
of end of semester 

Continuous during 
center operation 

Twice each semester 



Last two weeks 
of semester 

Before the beginninf 
of center operation 



Last two weeks of 
center operation 



GOAL II To expand educational opportunities in the Los Angeles Community College 
District by increasing college accessibility. 



TARGET 



TASK 



MEASURE 



DATE 



2.2 Thd Director 



2.8 Tht Bvaluator 



Will ensure admission 
priority as determined 
by criteria format 

Will determine the 
financial iiopact on the 
Community College 
District 

25* 



File of ranked applica- 
tions in Director's 
office. 

Evaluation report in 
Evaluator's office^ 



Continuously 
during center 
operation 

Within 2 weeks 
following end of 
semester 
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Seetlon 5 - Prowa Bwliutlon P#rformance Oblsctlvf Bttentlally Research 
Hypotheses ^ • 

It Is the present point of vlev of the three Los Angeles County Educational 
Program Auditors that data collected relative to performance objectives should be 
treated statistically In the same manner as data collected In the testing of 
research hypotheses* 

An example or two may be in order. Suppose an objective stated that there 
will be a 7-month gain (on Test A) pre to post (October and May) for students 
in the Xyz program* Since the auditor selects a sample of XYZ students » rescores 
pre and post tests, and computes a mean gain based on a sample of pre and post test 
scores, the auditor should employ a confidence Interval to test whether or not the 
objective was attained* Similarly, It is recognized that if the external auditor's 
sample mean or sample proportion is* to be compared to the internal eyaluator-s 
population mean or population proportion, the confidence interval method should 
again be used* - 

The confidence interval estimate procedures for the pre to post test gain and 
post test only are shown below in Tables 3 and 4* 



TABLE 3 - Confidence Interval Estimation Pormulae for the Pre to Post 
Test Gain Condition for Means or Porportions 





Assumptkmt 


- m - 

it filM 


where r-Af^l 




1. Dependence between 

Mmplcs 
1 Independence within 

amples 
y Normality 
4. Variances unknown 


or 






t . Independence between 

samples 
2. Independence within 

samples 
y Itinomul 
4. A',/»« > 5.AV/>o >5 



^r* James Vogler contributed to this section* 
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TABLE 4 - Confidei^^^ 

Only Co nditi o n for a Mea n o r P orportion 



Hypothesis Test statistic ] Confidence interval j Assumptions 


-Mo 


Sl^N _ 
whcrci'- (A^- I) 






1. Independence 

2. Normality 

3. Variance unknown 


Hoip^Po 
Mt'.p^Po 


^'pog^is 


P'P±Z\ 




1. Independence 

2. Binomial variable 

3. Np>S 
Sg>5 



The Confidence Interval Estimate Procedure was used last year. The results 
of one such analysis is shown in Table 5 below. 

Objeotive 4.0 . 

Upon completion of the program year in 1974, Title I project 
varbicipants wilh ^« spelling eUlls^ have demnstated wore than one 
months gain for eaoh month in the program as measured by scores on the WHAT. 



TABLE 5 Wide Range Achievement Test - Spelling 



Grade 


Sample 


Means 
Raw Score 


Gnide 
Equivalents 


Gains in 
Months 


Attainment 
of Obiective 


Concurrence 






Pre Post 


Pre 


Post 




Yes 


No 




K 


10 


7.2 18.0 


PK7 


1.0 


13 


✓ 




Concur 


1 


10 


IS.O 25.0 


K7 


1.5 


8 






Concur 


2 


10 


19.4 2S.0 


1.1 


2.2 


11 


Y 




Concur 


3 


10 


26.3 31.0 


1.8 


2.6 


8 






Concur 


4 


10 


28.7 35.5 


2.3 


3.2 


9 


~ v' 




Concur* 


5 


10 


30.1 35.2 


2.5 


3.2 


7 






Concur* 


6 


10 


35.4 40.7 


3.2 


4.7 1 15 






Concur 



*An upper statistical limit score was calculated and used for grades four 
and five using the t distribution and the- confidence interval method. 

The Auditor concurs with the evaluator that the objectives for grades 
kindergarten through sixth grade were attained. . 
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for th« data frooi gradaa 4 and 5| the auditor vouldjiave dleagreed with the 
evaluator*e etatenent that the objective had been attalnedi had he not calculated 

the confidence Interval. 

Section 6: Sotut Quest lona and Anavers Regarding the Educational Program Audit 
2 

Process 

WHAT IS AN EDUCATIONAL PROGRAM AUDIT? 

DEFINITION: An educational program audit le a p«rforaance control process based upon 
external reviews conducted by qualified outside consultants. It la designed to verify 
the results of the evaluation of an educational program and to assess the appropriate* 
ness of evaluation procedures used for determining the effectiveness of the operation 
and management of the program* 

WHY HAVE AN AUDIT? WHAT MIGHT RESULT PROM IT? 

There are many reasons why local educational agencies request program audits. Some 
of the results of past audits are: 

1. The identification of clearer definitions of program personnel responsibilities. 
Z The gathering of needed baseline data on the level of success of an educational 
program. 

3. The determination of test scoring discrepancies. 

4. The Identification of model or "tumkey'Vprograms which deserve to be developed 
and expanded. 

5. The Identification of areas for inservice training. 

6. The enhancing of credibility for the ftogtm and for the local educational 
agency. 

^Most of the credit for the question and answer format and content of this section 
belonf^s to Dr. Jean Wiener. Other consultants who have contributed to thin fsoction 
Include Dr. Dale Russell and Dr. Tom Bishop. 
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WHAT ARE THE PROFESSIONAL ROLES OF THE EDUCATIONAL PROGRAM AUDITOR? 
The educational program auditor brings a relatively new professional role to 
education. He serves as a third party, free of local ties and Interests, who 
verifies the results of the Internal evaluation of an educational program. 

THE AUDITOR IS ! 

1. Independent from the program to be audited. 

2. A reviewer and not a declslon-roaker. 

3. A reporter and an observer. 

A. A prof esslonal who exercises care and Integrity In performing the audit 

examination and In preparing audit reports. 
5. Equipped to Innovate within his own profession and to encourage reforms 

In the schools. 

WHAT ARE THE STAGES IN THE AUDIT PROCESS? 

STEP A: Optional Pre-Audlt Assistance: Educational Program Evaluation Assistance 

Local educational agencies may elect to receive Educational Program Evaluation 
Assistance prior to having an audit of a program. Educational Program Evaluation 
Assistance is a service which is intended to strengthen the internal process of 
evaluation in a local educational agency. In collaboration with a county consultant 
the local evaluator designs an effective evaluation system with appropriate sta- 
tistical procedures which includes development of performance objectives, process 
objectives, evaluation specifications, the evaluation design, and related documents. 
Step 1 - Orientation 

Orientation sessions to the audit may be conducted for local educational agency 
personnel. 
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Step 2 - Review of the Total Prortram 

The product and process objectlvesi the evaluation specifications, design 
management plan, time frame, and related documentation are reviewed to ensure 
the audltabllity of the program* 

Step 3 - Development of the A udit Pla n 

The audit plan Is a planning and operational control document for the auditor 
and a quality and management control document for both the local prograta director 
and the local program evaluator* This plan Indicates the techniques, schedules, 
processes, and procedures which the auditor will use In Judging the adequacy of 
the evaluation process and In verifying the evaluation findings • One method of 
random sampling frequently used Is matrix sampling. 

Step A ~ The Audit Contract 

The audit contract Is a written agreement between the local education agency and 
the Los Angeles County Superintendent of Schools Office. The purpose of the 
contract Is to make explicit the conditions of the audit agreement. One section, 
for example. Is the Assurance of Confidentiality section. 

Step 5 - On-site Visit (s) 

Typically, the auditor makes one announced on-site visit and one unannounced 
on-site vlst. During on-site visits, the auditor collects firsthand data as 
Identified In the audit plan and for the purpose of ascertaining the degree of 
attainment of process objectives. 

Step 6 - Interim and Final Audit Reports 

Typically, the auditor presents, an Interim report following each on-site visit 
and a final audit report following the receipt of the final evaluation report. 
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- The final audit report is presented to the program evaluator, program director, 
and superintendent, usually XvriLhin twenty v;orkins dayn of receipt of the final 
evaluation report. All of the reports arc confidential and presented to only 
the program evaluator, program director, and superintendent. 

^tej)_7_:7_JR>dtJ^ 

Program Audit Servi ce T hey Have Received 

It is reco^,nized that the audit process should itself be evaluated by those x^yho 
receive the service • 

CAN LOCAL EVA LUATORS RILCEIVE ASSISTANC E? 

Yes . Should a local evaluator of a program which is being audited desire consultant 
assistance in the development of an evaluation management plan, statistical data 
analysis, data synthesis, evaluation report writing or interpretation, he should, 
as soon as possible, initiate a request for assistance by writing to Dr. Gordon 
Footman, Director, Division of Program Evaluation, Research, and Pupil Services, 
Los Angeles County Superintendent of Schools, 9300 East Imperial Highway,. Dovmey, 
CA 90242. ' ' 

CAN-PR0GRAI4 OBJECTIVES BR REVISED ? 

Yes, upon the agreement between the local program personnel and the auditor. 
WHAT AUDIT TRAIL D0CUI4E NTS SHOULD PR OGRAM PER SONNEL KEEP? 

In order to do the job of verifying the evaluation of an educational program, the 
auditor must have direct access to firsthand evaluation data. The rule here is 
"Keep all evaluation trail documents. Keep whatever tnensure is used by the evalu- 
ator to determine whether or not each program objective is attained." .Evaluation 
trail documents often used are test booklets and answer sheets, pre- to post- tests. 



31 



JPage 27 

or post-tests only, summary sheets, lavels mastered on criterion referenced tests, 
attendance rosters for inscrvice staff meetings • 

Whereas the Stull Bill is concerned with the evaluation of a single person, the 
audit is concerned with verifying the evaluation of an educational program. The 
local program evaluator will present a final evaluation report, and possibly interim 
reports as well, on the attainment or nonattainraent of program objectives. As an 
outside, third party, the auditor samples evaluation data and verifies the result*^ 
of the internally conducted program evaluation. The auditor's reports include . 
audit findings relative to the attainment or nonattainment of program ob:]ectives. 
\fliile the Stull Bill is concerned with a single person, the audit is concerned with 
a program. No person is mentioned by name in an audit report. If a program process 
obiective were stated as '*A11 instructors teaching a certain level will receive the 
math textbook, IJajyijgma^^^ by the first day of class," and if during 

the on-site visit the auditor found that .four instructors out of ten interviewed 
did not receive the books by the date indicated, the audit renort would present 
just this information. The point of focus for the audit is the program. 

wHATjtF jyn^^ 

Call your auditor of record to discuss the situation. 

The purpose of this paper and oral presentation is to give visiting CERA 
Conference attendees one consultant's view of the State of the Art and Science of 
Educational Program Evaluation Services (K-14) in I^s Angeles County. The three- 
way program evaluation services that have been provided to more than AO elementary 
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* 

secondary, and unified r^chool districts ancj community colleges are described ♦ 
Some of the results or the progress of evaluations of Early Childhood Education, 
Conpensatory Education, Mentally Gifted Minors, Educationally Handicapped, 
Career Awareness, Veterans* Recruitment, Children's Centers, and Ransom Reading 
Programs were examined; The assertion is made that performance objectives 
as used in program evaluations are essentially research hypotheses and that 
data analyses and interpretations should recognize this fact. Finally, the 
writer would like to agree v/ith Stufflebeam in his assertion that three criteria 
should be used in judging program evaluation: (1) standards of behavioral 
science research, (2) utility, and (3) cost-benefit considerations. 
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UCL/V CBFTRR FOR TllE STLQY OF riVAUJATIOri 
B/AUlATIOn r'ODEL 



Evaluation is the process of determining the kinds of decisions that have to be made and selecting, 
collecting, and interpreting the information needed in making these decisions. 



Five evaluation phases provide the framework for the decision-making process: 



NEEDS 
ASSESSMENT 



PROGRAM 
•PLANNING 



IMPLEMENTATION 
EVALUATION 



PROGRESS 
EVALUATION 



OUTCOME 
EVALUATION 



Ne<fds AsscKsinent involves stating the objec- 
tives to be met and d»»tormining how well an 
existing program is nteoUng thuse objectives. 
This information is used to identify school or 
program needs. 



In Program Planning* the evatuator provides the 
project director with tools to help make plan- 
ning decisions. He also builds into the program 
the procedures that will be nee^«»d for assessing 
ivhether*or not it Is operatin(^ as planned and 
how well it Is acht**:ytng its o*ljectiv^es. 



Implementation Evaluation is a monitoring 
process to determine the extent to which the 
specified elements of the program have been 
implemented as planned. 



Progress Evaluation provides information about 
the progress of the program*s components in 
meeting the program's objectives. This informa- 
tion is used to modify the program where 
ncce.sjwiry. 



Outcome Evaluation provides information 
about the success of the entire program. l*his 
Information can support a decision to maintain* 
modify* e.\pand« or discontinue the program. ~ 
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LOCAL EDUCATIONAL AGENCIES IN THE LOS ANGELES COUNTY INCLUDE THE FOLLOWING: 



I^ Elementary School Dlstric t s 



AlhaiDbra City 
Castaic Union 
Eastside Union 
East Whittier City 
El Monte 
Garvey 
Gorman 
Hawthorne 

Hermosa Beach City 
Hughes-Elizabeth Lakes Union 
Keppel Union 



Lancaster 

Lawndale 

Lennox 

Little Lake City 

Los Nictos 

Lowell Joint 

Manhattan Beach City 

Mountain View 

Newhall 

Palmdale 

Redondo Beach City 



Rosemead 
San Gabriel 
Saugus Union 

Soledad-Agua Dulce Union 
South Wiittier 
Sulphur Springs Union 
Valle Lindo 
Westside Union 
l^ittier City 
Wilsona 
Wiseburn 



II. Hiph S chool Districts 

Alhambra City 
Antelope Valley Union 
Cent incla Valley Union 



El Monte Union 
Hart; Wm. S/, Union 



South Bay Union 
WHittier Union 



III. Unified Schoo l Distr icts 

ABG Unified 
Arcadia Unified 
Azusa Unified 
Baldwin Park Unified 
Basset t Unified 
Bellflower Unified 
Beverly Hills Unified 
Bonita Unified 
Burbank Unified 
Charter Oak Unified 
Claremont Unified 
Comnton Unified' 
Cdvina-Valley Unified 
Culver City Unified 



Downey Unified 
Duarte Unified 
El Rancho Unified 
El Segundo Unified 
Glendale Unified 
Glendora Unified 
Hacienda-La Puente Unified 
Inglewood Unified 
La Canada Unified 
Las Virgenes Unified 
Lons Beach Unified 
Los Angeles Unified 
Lynwood Unified 
Monrovia Unified 



Montebello Unified 

Norwalk-La Mirada Unified 

Palos Verdes Peninsula Unified 

Paramount Unified 

Pasadena Unified 

Pomona Unified 

Roxi7land Unified 

San Marino Unified 

Santa Monica Unified 

South Pasadena Unified 

Temple City Unified 

Torrance Unified 

Walnut Unified 

West Covina Unified 



TV^ CotTOunltv Collcjics 

Antclone Valley 
Ccrritos 
Citrus 
Compton 

East Los Angeles 
El Camino 
Glendale 



Long Beach 

Los Angeles City 

Los Angeles Harbor 

Los Angeles Pierce 

Los Angeles Southwest 

liOs Angeles Trade-Technical 

Los Angeles Valley 



Ht* San Antonio 
Pasadena 
Rio Hpndo 
Santa Clarita 
Santa Monica 
West I-os Angeles 
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. APPENDIX C 



LOS mms mm mmmm\{ of schools 



Administrative Organization to 
facilitate delivery of Educational 
Program Evaluation Services to all 
School Districts and Community 
Colleges In Los Angeles County • 




COUNTY BOARD OF EDUCATION 

Dr. Daniel L. Towlcr, President 
Vincent H. Simpson, Vice President 
Robert M. Bo.ck, Member 

Mrs. Pauline Chase » Member . • 

Mrs. Lucille M. Fields, Member 
Dr. E. V. Pullias, Member 
Mrs. Kathryn Vanderhook, Member' 

Dr. Richard M. Cloves, County Superintendent of Schools 

Dr. E. May Ion Drake, Assistant Superintendent, Educational Programs 
and Services 

Dr. Gordon Footman, Director, Division of Program Evaluation, Research, 
and Pupil Services 

Dr. Warren Newman, Assistant Director, Division of Program Evaluation, 
Research, and Pupil Services 



37 



APPENDIX D 



Page 33 



List of programs where services have been provided by consultants in 
educational program evaluation and educational program audit • 



1. Administrator Evaluation 
2« Career Awareness 

3. Career Education 

4. Cliildren^s Centers (Community 
Colleges) 

5. Social Studies ^ Urban Planning 

6. Conputer Assisted Instruction 

7. Conputer Management 

8. Continuous Progress System in 
Mathematics 

9. Counseling 

10. Counseling Needs Assessment 

11. Criterion Reference Measurement 
System 

12« Decentralized Educational Plan 
or Management Plan 

13. Demonstration Reading Project 

14. District Math Program (3-4) 

15. ESEA Title i 

16. Early Childhood Education 

17. Educational Opportunity Program 
Services (Conmunity Colleges) 

18. Educationally Mentally Retarded 

19. Elementary ^bth 

20. Emotionally Handicapped 



21. English 

22* Funded Projects 

23. Graphic Arts 

24. Instructional Plan (Junior 
High) 

25. Learning Skills Center 

26. Mathematics, Individualized 
tomputational Skills Program 

27. Mentally Gifted Minors 

28. Nursing 

29. Opportunity Classes 

30. Reading 

31. Reading, Methods, and Techniques 

32. Ranedial English 

33. Ranedial Reading 

34. Science (Elementary) 

35. Semi-Departmentalized 
. Program (7-8) 

36. Special Education 

37. Staff Differentiation 

38. Teacher Inservice 

39. Team Teaching Assessment 

40. Veteran *s Recruitment 

41. Vocational Education 
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APPENDIX E ^ 



This is an alphabetical listing of the Los Angeles County Superintendent 
of Schools consultants who provide various kinds of assistance in Program Eval- 
uation* These consultants work under the direction of Dr. Richard Clowes - 
Superintendent » Dr. Maylon Drake - Assistant Superintendent > Dr. Gordon Footman - 
Division Director^ and Dr. Warren Newman - Assistant Division Director, Division 
of Program Evaluation, Research, and Pupil Services. 

Dr. Tom Bishop - Educational Program Evaluation Planning and Audit Assistance 
Dr. Tom Butterworth - Educational Program Evaluation Planning Assistance 
Mrs. Mary Hearne - Educational Program Evaluation Planning Assistance 
Dr. Don Kester - Educational Program Audit and Evaluation Planning Assistance 
Dr. John Martois - Direct Evaluator Assistance 

Dr. Jerry Olson - Educational Program Evaluation Planning and Audit Assistance 

Mr. Earl Oirrens - Direct Evaluator Assistance 

Dr. Dale Russell - Educational Program Audit Assistance 

Dr. Jim Vogler - Educational Program Evaluation Assistance 

Dr. Jean Wiener - Educational Program Audit Assistance 
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